1. Field of the Invention
The present invention relates generally to stream processing and, in particular, to workflow composition. Still more particularly, the present invention provides a method, apparatus, and program product for stream processing workflow composition using automatic planning.
2. Description of the Related Art
Stream processing computing applications are applications in which the volume of data being processed is too large to be stored and, therefore, the information must be processed on the fly. Examples of stream processing computing applications include video processing, audio processing, streaming databases, and sensor networks. Recent advances in software development technologies, together with the growing availability of powerful computers and high-bandwidth communications have brought to practice large compositional computing architectures.
This approach enables increased code reuse, simplified development, and high flexibility of the system. Components may be interconnected in multiple configurations, achieving highly complex functionality via composition of simpler black-box operations. Such architectures are being currently developed in many application areas, in particular, stream processing applications.
In compositional stream processing architectures, the stream processing applications are composed of several processing units or components. The processing units can receive information streams on one or more input ports and produce one or more output streams, which are sent out via output ports. The output streams are a result of processing the information arriving via the input streams, by filtering, annotating, or otherwise analyzing and transforming the information. Once an output stream is created, any number of other components can read data from it. All processing units together compose a workflow. A stream processing application reads and analyzes primal streams coming into the system and produces a number of output streams that carry the results of the analysis.
Composing stream processing workflows is a labor-intensive task, which requires that the person building the workflow has an extensive knowledge of component functionality and compatibility. In many cases, this makes it necessary for end-users of stream processing applications to contact application developers each time a new output information stream is requested and, as a result, a new workflow is needed. This process is costly, error-prone, and time-consuming. Also, changes to other elements of the stream processing system may require changes to the workflow. For example, processing units or primal streams may become unavailable, users may place certain restrictions on the output, or changes may be made to the components themselves.
In large practical stream processing systems, both changes in the data coming into the system and changes in the system configuration can invalidate deployed and running stream processing applications. With time, these applications can start to produce output that no longer satisfies the user's requirements or they can be relying on primal streams that have become inactive. In many situations, user's requirements can be better satisfied if an existing workflow is updated with newly available primal streams or processing units. Therefore, when changes such as those described above occur, the workflow must be reconfigured quickly, before any potentially valuable streaming data is lost. Such timely reconfiguration is extremely difficult to achieve if the workflow composition requires human involvement.